Real-time detection of object scanability

ABSTRACT

A system and method are disclosed for determining and alerting a user as to whether an object will successfully scan before the post-processing of the scan data. In embodiments, before post-processing of the scan data begins, the scan data is processed by a machine learning algorithm which is able to determine whether and/or how likely the scan data is to return an accurate scanned reproduction of the scanned object. The machine learning algorithm may also suggest new positions for the object in the environment where the scan is more likely to be successful.

BACKGROUND

Three-dimensional (3D) scanning technologies use image capture devicesto scan a real world object and reproduce the object either on a displayor as a real world object replica using 3D printing techniques. A keyobstacle to widespread adoption of object scanning is that there arestill many objects that do not produce an accurate reproduction whenscanned by current technologies. There are a variety of reasons why aparticular object may not reproduce accurately when scanned, includingfor example specularity, reflectivity, surface textures, surfaceconcavities, etc.

This problem is compounded by the fact that conventional scanners do notalert the user when a target object cannot be scanned. By not knowingwhether an object is “scanable” or not, the user often invests asignificant amount of time in a scanning process that will ultimatelyfail. Post-processing of scan data can take up to an hour or two toreturn the scanned replica, with the user not being made aware that itwill not succeed until the post-processing step is completed.

SUMMARY

Embodiments of the present technology relate to methods of determiningand alerting a user as to whether an object will successfully scanbefore the post-processing of the scan data. In embodiments, beforepost-processing of the scan data begins, the scan data is processed by amachine learning algorithm, referred to herein as a scanabilityalgorithm, which is able to determine whether and/or how likely the scandata is to return an accurate scanned reproduction of the scannedobject. The scanability algorithm may also suggest new positions for theobject in the environment where the scan is more likely to besuccessful.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating a learning mode of a scanabilityalgorithm according to embodiments of the present technology.

FIG. 2 is a perspective view of a user scanning an object from a firstperspective.

FIG. 3 is a perspective view of a user scanning the object of FIG. 2from a second perspective.

FIG. 4 is a perspective view of a user scanning an object from a firstperspective.

FIG. 5 is a perspective view of a user scanning the object of FIG. 4from a second perspective.

FIG. 6 is an illustration of a scanability model implemented in amachine learning neural network according to an embodiment of thepresent technology.

FIG. 7 is a flowchart providing further detail of step 208 for trainingthe scanability model.

FIG. 8 is a block diagram of the operation of the scanability model ofthe present technology to predict the success or failure of a scanneddevice.

FIG. 9 is a flowchart of the operation of the present technology in useto predict the success or failure of a scanned object.

FIG. 10 is a flowchart showing additional detail of step 232 from FIG.9.

FIGS. 11-14 are illustrations of a head mounted display including apop-up menu with a prediction as to whether the scan will be successfulor unsuccessful.

FIG. 15 is a block diagram of an illustrative computing system forimplementing aspects of the present technology.

DETAILED DESCRIPTION

Embodiments of the present technology will now be described withreference to the figures, which in general relate to methods ofdetermining in real time whether a given object will successfully scanin a given environment before the post-processing of the scanned data.In embodiments, before post-processing of the scanned data begins, thescanned data is processed by a machine learning scanability model, whichis able to determine whether and/or how likely the scan data is toreturn an accurate scanned reproduction of the scanned object.

In embodiments, the scanability model is developed from a machinelearning algorithm which may be trained to recognize scan data that islikely to result in a successful scan versus an unsuccessful scan. Alarge number of learning trials may be performed where objects arescanned by one or more image sensors, and the scan data stream(s) areinput into the machine learning algorithm. The output of the machinelearning algorithm in a given trial is compared against the actualoutput from post-processing of the scan data, and from this, the machinelearning algorithm may be updated or refined based on any deviation fromthe respective outputs.

In general, the scanability algorithm may develop the model to increasethe accuracy of its predictions by performing a large number of learningtrials over which the scanability model is trained to make accuratepredictions. In embodiments, there may be between one thousand and onehundred thousand learning trials for the scanability model, but thenumber of learning trials may be less than one thousand and greater thanone hundred thousand in further embodiments. Thereafter, the scanabilityalgorithm may be used by end-users in performing scans. Using therefined scanability model, the scanability algorithm may inform a userof the likely outcome of an object scan before post-processing of thescan data.

In embodiments, in addition to notifying a user that a scan will likelybe unsuccessful, the scanability algorithm may also suggest newpositions for the object in the environment where the scan is morelikely to be successful. In embodiments, new example data can becomposited out of the existing captures generated by the scanabilityalgorithm to generate a suggestion of an alternate position. Forexample, during the scan of the object, a volume may be defined wherethe scan takes place. This volume, shown for example in the depth data,can then be synthetically re-injected onto other surfaces to try anddetermine if it might scan better in other parts of the environment.

Additionally, as explained below, the scanability model built by thescanability algorithm may continue to learn and be updated based onfeedback on the accuracy of the model when in use by end-userspredicting actual instances of a successful or unsuccessful scan.

The flowchart of FIG. 1 shows an embodiment of how the machine learningscanability algorithm of the present technology may generate thescanability model. In step 200, the scanability algorithm receives thescan data from a first learning trial. The scan data may comprise datastreams generated by one or more image sensors capturing images of anobject and surrounding environment from different perspectives. Inembodiments, the image sensor may include a depth camera generating adepth data stream, an RGB camera generating an RGB data stream and/or aninfrared (IR) camera generating an IR data stream. In any given trial,the scan data may be comprised of image data from one or more of thesedata streams. Other image sensors may also or alternatively be used. Asexplained below, pose data may be used as an indicator of whether thescan will be successful or unsuccessful.

In embodiments described below, the scan data is 3D scan data generatedfrom one or more image sensors of a head mounted display device forpresenting an augmented reality experience. Details of the operation ofthe image sensors of a head mounted display device are peripheral to thepresent technology, beyond the fact that they are able to capture imagesof an object from different perspectives using for example a depthcamera, an RGB camera and/or an IR camera. However, a description of ahead mounted display device and its image sensors is provided forexample in U.S. Patent Publication No. 2013/0326364 entitled “PositionRelative Hologram Interactions,” published on Dec. 5, 2013.

It is understood that the present technology is not limited to headmounted display devices or to receiving scan data from particular imagesensors. The present technology may receive scan data from a widevariety of scanning devices using a wide variety of different imagesensors. It may happen that a scanning device employs each of a depthcamera, RGB camera and an IR sensor, in which case each of these datastreams may be provided to the scanability algorithm for analysis.

However, it may also be that a scanning device employs less than allthree of these image sensors. It is a feature of the present technologythat the scanability algorithm and refined model can operate with datastreams from one or more image sensors, without having information onwhich image sensors are providing the data streams. In particular, asexplained below, data streams that are received are input to anartificial neural network machine learning algorithm. The algorithmperforms learning trials where one or more data streams, for examplecomprised of depth data, RGB data, IR data and/or pose data, arereceived. The algorithm gets trained to identify whether the one or moreinput data streams, in a given instance, will likely result in asuccessful scan or an unsuccessful scan.

The data streams received in step 200 are based on a scanning devicecapturing a real world 3D object from a variety of differentperspectives. For example, FIGS. 2 and 3 illustrate examples of a userwearing a head mounted display device 100 for scanning an object 102from different perspectives within an environment 104 having a surface106 on which the object 102 is situated. In embodiments, the environment104 scanned may be an area directly appurtenant to the object. Thus, inFIGS. 2 and 3 the environment 104 may consist entirely of the surface106.

The head mounted display device 100 may include one or more imagesensors including for example a depth sensor, RGB camera and/or an IRsensor, which capture image data from the object 102 and environment104. The scan data may then be processed by the scanability algorithmexecuting within a processing unit 108 associated with the head mounteddisplay device 100. In alternative embodiments, it is also possible toupload the scan data to one or more remote servers in the cloud and thenrun the scannability algorithm in the cloud. This may be useful forexample when the scanning device does not have enough computing power orspecial computing hardware to run the algorithm efficiently. FIGS. 2 and3 show the user capturing the object 102 from two differentperspectives, but it is understood that the user may move around theobject and capture the object from several more perspectives.

FIGS. 4 and 5 illustrate a user capturing scan data of an object 102within an environment using a hand-held computing device 110 includingone or more image sensors including, for example a depth sensor, RGBcamera and/or an IR sensor. The scanned data may then be processedwithin a processing unit within or associated with the computing device110 and executing the scanability algorithm. The computing device 110may for example be a laptop, tablet, smart phone or an image scannerassociated with a scanning device. Again, FIGS. 4 and 5 show the usercapturing the object 102 from two different perspectives, but it isunderstood that the user may move around the object and capture theobject from several more perspectives.

The object 102 may have been imaged from a sufficient number ofdifferent perspectives such that image data from all visible surfaces ofthe object is received. In this instance, the learning algorithm willdetermine whether all visible surfaces of the object may be successfullyrecreated in the scan, as explained below. However, it is understoodthat where a scanning device has scanned a 3D object from only limitedperspectives, for example a front half of an object, the learningalgorithm will determine whether the front half of the object (orwhatever views are available from the scanned perspectives) may besuccessfully recreated in the scan.

Depending on the scanning device and image sensor(s) used in a scan, thescan data will include a wide variety of information relating to theattributes of both the captured object and the environment in which theobject is captured. For example, the scan data may include data relatingto the following attributes of the object:

-   -   specularity, or specular reflection—light, reflecting in        parallel rays off of smooth surfaces of the scanned object,        which is received in the image sensor may make it difficult to        determine the nature of the surfaces;    -   diffuse reflections—light, reflecting diffusely off of rough        surfaces of the scanned object, which is received in the sensor        may make it difficult to determine the nature of the surfaces;    -   transparency—light, passing through surfaces of the scanned        object, may make it difficult to determine the nature of the        surfaces;    -   texture—topological characteristics of the object, relating to        the uniformity, roughness, waviness, etc. of surfaces of the        scanned object, may make it easier or harder to determine the        nature of the scanned surfaces;    -   concavities—surfaces which include concavities may be difficult        to scan accurately;    -   symmetry—objects which are symmetrical about one or more planes        may be more difficult to scan accurately;    -   localized cues—objects which do not have localized cues, such as        distinct lines, points or other discontinuities, may be more        difficult to scan accurately.        The above is not intended as a comprehensive listing of the        attributes which particular image sensors may capture, and it is        understood that a variety of other attributes may be gleaned        from the scanned data of the object in addition to or instead of        those listed above.

Each of the above attributes may affect whether an object may besuccessfully scanned. However, as explained below, embodiments of thescanability algorithm are able to develop a model which accounts forthese object attributes without having to individually quantify orclassify these attributes within the scan data.

In addition to the attributes of the object, the image sensor(s) maycapture data relating to the environment in which and on which theobject is situated. The environment, including a surface on which theobject is situated, may have the same types of attributes described forthe object above. An additional environmental attribute may beluminance; i.e., the amount of natural or man-made light in theenvironment. Too much luminance may result in high specular or diffusereflection off of the object or surface on which the object is situated,and make it difficult to accurately scan the object. Conversely, toolittle luminance may make it difficult accurately detect surfaces of theobject. Again, as explained below, embodiments of the scanabilityalgorithm are able to develop a model which accounts for theseenvironmental attributes without having to individually quantify orclassify these attributes within the scan data.

Once the one or more scan data streams have been received in step 200,post-processing of the scan data may be performed in step 202.Post-processing of the scanned data refers to the processing of theimage data to generate the scanned reproduction of the object. As notedin the background section, the post-processing step may be timeconsuming. However, during the learning trials, post-processing isperformed to determine whether a given set of one or more data streamsdoes or does not in fact result in a successful scan of the object.

In step 204, one or more quality metrics algorithms are run to determinewhether the scan was successful or how successful the scan was. Thequality metrics algorithm(s) may measure different characteristics ofthe scanned reproduction, and produce a quantitative or qualitativeoutput by which to just the success or failure of the scannedreproduction in a given trial. The quality metric may be a pass/failrating or a sliding scale numeric value, with for example a low numberrepresenting an accurate and successful reproduction and a high numberrepresenting an inaccurate failed reproduction. In embodiments, qualitymetrics algorithm(s) may be programmed to value certain reproducedcharacteristics in the scanned reproduction over other reproducedcharacteristics. Which reproduced characteristics are emphasized overothers in the quality metrics algorithm(s) may vary in differentembodiments, depending at least in part on what characteristics theadministrators of the quality metrics algorithms consider to beimportant.

In step 206, the quality metric(s) determined by the one or more qualitymetrics algorithm is/are assigned as ground truth, or an “objectiveoutput” for the one or more input data streams in each learning trial totrain the machine learning algorithm as explained below.

In step 208, the one or more input data streams and the objective outputare used to train the scanability algorithm and develop the finishedscanability model. Further details of step 208 are explained now withreference to the diagram of FIG. 6 and the flowchart of FIG. 7.

In general, the scanability algorithm employs an artificial neuralnetwork (ANN) 120, a symbolic illustration of which is shown in FIG. 6.ANN 120 is comprised of a number of neurons, or nodes, 122 (one of whichis numbered) which exchange information with each other as indicated bythe arrow connectors 130 (one of which is numbered). The nodes 122 maybe organized in layers, including an input layer 124, a hidden layer 126and an output layer 128. In embodiments, the scanability algorithm mayemploy many more layers in ANN 120 than are shown in FIG. 6. Eachconnector 130 may be assigned a numeric weight. Over the learningtrials, the numeric weights associated with each connector may berefined as explained below until the ANN 120 is developed into ascanability model which can accurately predict an outcome of asatisfactory scan or unsatisfactory scan for a given set of one or morescan input data streams.

Referring now to the flowchart of FIG. 7, the one or more input datastreams are input to the input layer 124 of the ANN 120 in step 218. Forexample, FIG. 6 shows a depth data input stream, an RGB data inputstream and an IR data input stream. However, as noted above, there maybe less than the three data streams provided to the input layer 124 andthere may be additional and/or alternative types of data stream that aresupplied to the input layer 124 in further embodiments. As one furtherexample, pose data may also be used as an input data stream to the inputlayer 124 of the ANN 120. In particular, where pose data is jittery ormoving at high speeds, this may be used as an input tending to show apoor result.

In step 220, the one or more input data streams may propagate across thenodes 122 and layers to the nodes 122 of the output layer. The input toeach node 122 may be processed as is known for conventional artificialneural networks and the output of each node is passed along a connector130 to the next layer, multiplied by the numerical weight of theconnector. The first time through ANN 120, the numeric weights may berandomly or arbitrarily selected, as they will be adjusted over thelearning trials. The result of the forward propagation of the one ormore input data streams will be a calculated output of either asuccessful scan or an unsuccessful scan.

In step 222, the scanability algorithm checks whether the outcome iscorrect. That is, it compares the calculated output for a given set ofone or more data streams against the objective output determined throughpost-processing of the one or more data streams in step 204 as describedabove. The objective output of the post-processing step may be providedto the scanability algorithm for use in step 222 by an administrator viaa user interface associated with the computing device executing thescanability algorithm.

Where the calculated outcome matches the objective outcome in step 222for a given trial, the scanability algorithm determined the correctoutcome, and the algorithm may proceed to the next learning trial. Onthe other hand, where the calculated outcome does not match the objectoutcome in step 222 for a given trial, the scanability algorithmdetermined an incorrect result. In this event, the numeric weights ofone or more of the connectors may be adjusted in a known manner in step226, using for example a back-propagation method. Other known methods ofadjusting the weights of the connectors may be used in furtherembodiments. The algorithm may then proceed to the next learning trial.

Referring again to FIG. 1, once the scanability model has been trainedwith the data for a given trial, the algorithm checks in step 210whether there are additional trials. In embodiments, this may be apredetermined number of trials. In further embodiments, the learningtrials may continue until the weights of the ANN 120 have been refinedto the point where the ANN 120 is returning a correct prediction of asuccessful or unsuccessful scan above in some predetermined percentageof the trials. This percentage may for example be between 95% and 100%,though it may include lower percentages in further embodiments.

Upon completion of the learning trials and tuning of the weights of theANN 120, the ANN 120 with the refined weights may be published as ascanability model 152 (FIG. 8) for use by end users in predicting theoutcome of a scan of an object. An example of an implementation of thescanability model 152, as used by an end-user performing a scan, willnow be explained with reference to the flowchart of FIG. 9.

Once the user scans an object, using for example any of theabove-described scanning devices, the scanability model 152 may receivethe scan data 150 in a step 230. In step 232, using the scanabilitymodel 152, the algorithm may determine whether an accurate (successful)scan will result from the scan data. Further details of step 232 willnow be described with reference to the flowchart of FIG. 10.

In step 260, the one or more data streams in the scan data may beprovided to the input layer of the ANN 120 of the scanability model 152.The ANN 120 of model 152 operates as described above to predict anoutcome. However, as the values of the weights are now presumablyrefined, the back-propagation tuning step of the weights is omitted.

In embodiments, the forward propagation of the input data streamsthrough the layers of the scanability model may strongly point to eithera success or failure of the scan. The strength to which the calculatedoutput of the scanability model points to one or the other of successfulscan/failed scan may be quantified into a confidence value in step 264.In step 268, the scanability model 152 may return a prediction of asuccessful or unsuccessful scan from the calculated output, togetherwith a confidence value representing a confidence in the calculatedoutput. Calculation of the confidence value in step 264 may be skippedin further embodiments.

Returning to the flowchart FIG. 9, in step 236, if the scanability modelpredicts a successful scan, the prediction may be communicated to theuser in step 238 and post-processing may be performed on the scan datain step 240.

FIGS. 11 and 12 show the head mounted display 100 including the object102 to be scanned. A prediction of a successful scan may be communicatedto the user for example by a pop-up display window 160 appearing in theuser's field of view. In FIG. 11, the prediction is communicated, and inFIG. 12, the prediction and confidence of the prediction iscommunicated. The prediction and/or confidence may be communicated tothe user a variety of other ways, including with audio, in furtherembodiments.

Post-processing of the scan data into a reproduction of the scannedobject may be performed by any of a variety of post-processingalgorithms. As indicated above, the present technology does not relateto the efficacy of the post-processing algorithm. The scanability modelof the present technology may be included as part of the post-processingalgorithm, or vice versa. The scanability model may be separate from thepost-processing algorithm in further embodiments.

At a completion of the scan, a user may have the ability to providefeedback as to the accuracy of the prediction in step 242. For example,if an accurate scan was predicted, but in fact the resulting scan wasnot accurate, a user may provide feedback via a user interface, whichfeedback may be used to update and further train the numerical weightsin the scanability model in step 246.

On the other hand, if the scanability model predicts a failed scan instep 236, the model may communicate this prediction to a user in step248 and as shown in FIGS. 13 and 14. Where post-processing of scan datato generate a reproduction of the scanned object may take a long time,for example on the order of 1 to 2 hours, the prediction as to asuccessful or failed scan in accordance with aspects of the presenttechnology may be determined in real time; that is, nearlyinstantaneously by a processor executing the scanability model in forexample a few milliseconds. Thus, the prediction as to whether the scanof an object will be successful or unsuccessful may be provided beforethe post-processing of the scan data begins. In further embodiments, theprediction as to whether the scan of an object will be successful orunsuccessful may be provided just after the post-processing of the scandata begins and long before the post-processing is completed.

Upon receiving the prediction of a failed scan in steps 236 and 248, auser may abort post-processing of the scan data in step 252, and therebysave a significant amount of otherwise wasted time. Of course, uponreceiving the prediction of a failed scan, the user may elect to proceedwith the scan anyway, and step 252 may be skipped.

In accordance with further aspects of the present technology, uponreceiving an indication of a failed scan, scanability model may suggestto the user in step 254 ways in which the scan may be modified that willresult in a successful scan. For example, if too much light within aportion of the environment may make scanning difficult. As a furtherexample, where the attributes of the object (reflectivity, transparency,etc.) are too similar to the attributes of the surface on which theobject is supported, this may make scanning of the object difficult.

In accordance with this embodiment, the scanability model 152 may detectthese attributes in the one or more of the input data streams(independent of the ANN 120). Thus, where for example the model detectsa high degree of luminance, such as for example where the object is indirect sunlight, the model may suggest to the user that the objectremoved to an area of the environment with less luminance. As a furtherexample, where the attributes of the object and a surface on which theobject is located are too similar, the model may suggest that the usermove the object to a different surface. Other examples are contemplated.

FIGS. 13 and 14 show the head mounted display 100 including the object102 to be scanned. A prediction of an unsuccessful scan may becommunicated to the user for example by a pop-up display window 160appearing in the user's field of view. In FIG. 13, the prediction iscommunicated, and in FIG. 14, the prediction is communicated togetherwith a suggestion of how to reposition the object to result in betterscan. The prediction and/or suggestion may be communicated to the user avariety of other ways, including with audio, in further embodiments.

FIG. 15 described below sets forth a computing device including aprocessor for executing the scanability algorithm and memory for storingthe inability model. In one embodiment, during the learning trials inthe development and training of the scanability model, the scanabilityalgorithm may run on one or more servers under the control of an entityadministering the scanability algorithm. Once the learning trials havebeen completed, the scanability algorithm and scanability model may bedistributed to client computing devices, including a head mounteddisplay device or other computing device, to receive predictions whenscanning an object. In such an embodiment, when feedback on a successfulscan prediction is provided (step 242, FIG. 9), that feedback to beuploaded from the client device to a central server.

The central server may in turn update the scanability model and thendistribute the updated scanability model to client devices using thescanability algorithm.

In a further embodiment, the scanability algorithm may be implementedwithout the learning trials being performed on the one or more serversunder the control of the administering entity. In such an embodiment,the learning trials may in effect be crowd sourced to individual userson their respective client devices. Each such use may result in aprediction, and the users may provide feedback on those predictions.That feedback may be uploaded from the client devices to a centralserver. The central server may in turn update the scanability model andthen distribute the updated scanability model to the respective clientdevices for continued use, training and development of the scanabilitymodel.

FIG. 15 illustrates a computing system 300 for executing the scanabilityalgorithm and for storing the scanability model described above. Thecomputing system 300 is one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the present system. Neither should the computing system300 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplarycomputing system 300.

The present system is operational with numerous other general purpose orspecial purpose computing systems, environments or configurations.Examples of well-known computing systems, environments and/orconfigurations that may be suitable for use with the present systeminclude, but are not limited to, personal computers, server computers,multiprocessor systems, microprocessor-based systems, network PCs,minicomputers, hand-held computing devices, mainframe computers, andother distributed computing environments that include any of the abovesystems or devices, and the like. In the distributed and parallelprocessing cluster of computing systems used to implement the presentsystem, tasks are performed by remote processing devices that are linkedthrough a communication network. In such a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 15, an exemplary computing system 300 for use inperforming the above-described methods includes a general purposecomputing device. Components of computing system 300 may include, butare not limited to, a processing unit 304, a system memory 316, and asystem bus 321 that couples various system components including thesystem memory to the processing unit 304. The system bus 321 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. By way of example, and not limitation, sucharchitectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus also known as Mezzanine bus.

The system memory 316 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 331and random access memory (RAM) 332. A basic input/output system (BIOS)333, containing the basic routines that help to transfer informationbetween elements within computing system 300, such as during start-up,is typically stored in ROM 331. RAM 332 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 304. By way of example, and notlimitation, FIG. 15 illustrates operating system 306, applicationprograms 310, other program modules 336, and program data 337.

The computing system 300 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 15 illustrates a hard disk drive 341 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 351that reads from or writes to a removable, nonvolatile magnetic disk 352,and an optical disk drive 355 that reads from or writes to a removable,nonvolatile optical disk 356 such as a CD-ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,DVDs, digital video tape, solid state RAM, solid state ROM, and thelike. The hard disk drive 341 is typically connected to the system bus321 through a non-removable memory interface such as interface 340, andmagnetic disk drive 351 and optical disk drive 355 are typicallyconnected to the system bus 321 by a removable memory interface, such asinterface 350. In the context of this document, a computer readablestorage medium may be any tangible medium that can contain, or store aprogram for use by or in connection with an instruction executionsystem, apparatus, or device. A computer readable storage medium doesnot include transitory, modulated or other types of signals.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 15 provide storage of computer readableinstructions, data structures, program modules and other data for thecomputing system 300. In FIG. 15, for example, hard disk drive 341 isillustrated as storing operating system 344, application programs 345,other program modules 346, and program data 347. These components caneither be the same as or different from operating system 306,application programs 310, other program modules 336, and program data337. Operating system 344, application programs 345, other programmodules 346, and program data 347 are given different numbers here toillustrate that, at a minimum, they are different copies.

A user may enter commands and information into the computing system 300through input devices such as a keyboard 362 and pointing device 361,commonly referred to as a mouse, trackball or touch pad. Other inputdevices (not shown) may be included. These and other input devices areoften connected to the processing unit 304 through a user inputinterface 360 that is coupled to the system bus 321, but may beconnected by other interface and bus structures, such as a parallelport, game port or a universal serial bus (USB). A monitor 391 or othertype of display device is also connected to the system bus 321 via aninterface, such as a video interface 390. In addition to the monitor391, computers may also include other peripheral output devices such asspeakers 397 and printer 396, which may be connected through an outputperipheral interface 395.

As indicated above, the computing system 300 may operate in a networkedenvironment using logical connections to one or more remote computers inthe cluster, such as a remote computer 380. The remote computer 380 maybe a personal computer, a server, a router, a network PC, a peer deviceor other common network node, and typically includes many or all of theelements described above relative to the computing system 300, althoughonly a memory storage device 381 has been illustrated in FIG. 15. Thelogical connections depicted in FIG. 15 include a local area network(LAN) 371 and a wide area network (WAN) 373, but may also include othernetworks. Such networking environments are commonplace in offices,enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computing system 300 isconnected to the LAN 371 through a network interface or adapter 318.When used in a WAN networking environment, the computing system 300typically includes a modem 372 or other means for establishingcommunication over the WAN 373, such as the Internet. The modem 372,which may be internal or external, may be connected to the system bus321 via the user input interface 360, or other appropriate mechanism. Ina networked environment, program modules depicted relative to thecomputing system 300, or portions thereof, may be stored in the remotememory storage device. By way of example, and not limitation, FIG. 15illustrates remote application programs 385 as residing on memory device381. It will be appreciated that the network connections shown areexemplary and other means of establishing a communications link betweenthe computers may be used.

In summary, embodiments of the present technology relate to a method,comprising: (a) receiving scan data from a plurality of scans of one ormore objects in a plurality of trials; (b) one of determining orreceiving objective outputs in the plurality of trials as to whether thescan data results in a successful or failed scan of the one or moreobjects in the plurality of trials; (c) processing the scan data in theplurality of trials using an algorithm to determine calculated outputsin the plurality of trials as to whether the scan data results in asuccessful or failed scan of the one or more objects in the plurality oftrials; and (d) refining the algorithm over the plurality of trials toimprove a correspondence between the calculated output and the objectiveoutput where the calculated output differs from the objective output forthe same scan data, the algorithm operable to predict a success orfailure of scans of objects performed upon completion of the pluralityof trials

In further embodiments, the present technology relates to a computingdevice, comprising: one or more scanners comprising one or more imagesensors operable to scan image data from a plurality of objects in anumber of trials; and a processor operable to: receive, in each of thenumber of trials, scan data from the one or more image sensors, andgenerate a model, trained using the scan data from the one or more imagesensors in the number of trials, enabling real time predictions of asuccess or failure for scan of an object performed upon completion ofthe number of trials.

In other embodiments, the present technology relates to a computerreadable medium for storing computer instructions for providing feedbackon whether a scan of an object will be successful, with the computerinstructions executed by one or more processors to perform the steps of:(a)receiving scan data from the object, the scan data comprising one ormore data streams; and (b) predicting whether the scan of the objectwill be successful or unsuccessful based on the one or more data streamsof the scan data, said step of predicting occurring beforepost-processing of the scan data to obtain a reproduction of the scannedobject is completed.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims. It is intended that the scopeof the invention be defined by the claims appended hereto.

We claim:
 1. A method, comprising: (a) receiving scan data from aplurality of scans of one or more objects in a plurality of trials; (b)one of determining or receiving objective outputs in the plurality oftrials as to whether the scan data results in a successful or failedscan of the one or more objects in the plurality of trials; (c)processing the scan data in the plurality of trials using an algorithmto determine calculated outputs in the plurality of trials as to whetherthe scan data results in a successful or failed scan of the one or moreobjects in the plurality of trials; and (d) refining the algorithm overthe plurality of trials to improve a correspondence between thecalculated output and the objective output where the calculated outputdiffers from the objective output for the same scan data, the algorithmoperable to predict a success or failure of scans of objects performedupon completion of the plurality of trials.
 2. The method of claim 1,further comprising the step of the algorithm suggesting options forrepositioning an object scanned unsuccessfully in the scans of theobjects performed upon completion of the plurality of trials.
 3. Themethod of claim 1, further comprising the step of receiving feedback onthe prediction of the success of the scan of the object performed uponcompletion of the number of trials, and updating the algorithm based onthe feedback.
 4. The method of claim 1, said step of refining thealgorithm over the plurality of trials comprising the step of refiningnumerical weights used by the algorithm to improve a correspondencebetween the calculated output and the objective output.
 5. The method ofclaim 1, the algorithm operable to predict, in real time, a success orfailure of scans of objects performed upon completion of the pluralityof trials.
 6. A computing device, comprising: one or more scannerscomprising one or more image sensors operable to scan image data from aplurality of objects in a number of trials; and a processor operable to:receive, in each of the number of trials, scan data from the one or moreimage sensors, and generate a model, trained using the scan data fromthe one or more image sensors in the number of trials, enabling realtime predictions of a success or failure for scan of an object performedupon completion of the number of trials.
 7. The computing device ofclaim 6, wherein the processor is further operable to suggest optionsfor repositioning the object scanned unsuccessfully in the scan of theobject performed upon completion of the number of trials.
 8. Thecomputing device of claim 6, wherein the processor is further operableto perform a post-processing operation, in a trial of the number oftrials, to reach an objective output as to whether the scan datareceived for that trial successfully produces a reproduction of anobject.
 9. The computing device of claim 8, wherein the processor isfurther operable to generate the model from training based on the scandata from the one or more image sensors in the trial and the objectiveoutput in the trial.
 10. The computing device of claim 6, wherein theprocessor is further operable to perform post-processing operations, ineach trial of the number of trials, to reach a number of objectiveoutputs, one objective output in each trial, as to whether the scan datareceived for each trial successfully produces a reproduction of anobject.
 11. The computing device of claim 10, wherein the processor isfurther operable to generate the model from training based on the scandata from the one or more image sensors in each trial and the objectiveoutput in each trial.
 12. The computing device of claim 11, wherein theprocessor is further operable to process the scan data through the modelin each trial to determine a calculated output in each trial, and tuningthe model based on a similarity or dissimilarity between the calculatedoutput and the objective output.
 13. The computing device of claim 11,wherein the processor implements an artificial neural network includingnumeric weights that are refined in the number of trials based on asimilarity or dissimilarity between the calculated output and theobjective output in each trial.
 14. The computing device of claim 6,further comprising the step of receiving feedback on the prediction ofthe success of the scan of the object performed upon completion of thenumber of trials, and updating the model based on the feedback.
 15. Thecomputing device of claim 6, wherein the computing device is used inassociation with a head mounted display device providing an augmentedreality experience.
 16. A computer readable medium for storing computerinstructions for providing feedback on whether a scan of an object willbe successful, with the computer instructions executed by one or moreprocessors to perform the steps of: (a) receiving scan data from theobject, the scan data comprising one or more data streams; and (b)predicting whether the scan of the object will be successful orunsuccessful based on the one or more data streams of the scan data,said step of predicting occurring before post-processing of the scandata to obtain a reproduction of the scanned object is completed. 17.The computer readable medium of claim 16, wherein the one or more datastreams are comprised of one or more of depth data, RGB data, infrareddata and pose data.
 18. The computer readable medium of claim 17,wherein said step (b) of predicting whether the scan of the object willbe successful or unsuccessful occurs regardless of which of the one ormore data streams are received.
 19. The computer readable medium ofclaim 16, wherein said step of predicting occurring beforepost-processing of the scan data to obtain a reproduction of the scannedobject is started.
 20. The computer readable medium of claim 16, furthercomprising the step of receiving an indication to abort the postprocessing of the scan data upon receiving a prediction that the scanwill not be successful.